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ABSTRACT 

During the 1990s, the use of meta-analytic methods in 
educational research has been widespread, and few aspects of education have 
escaped the meta-analytic revolution. The acceptance has not been complete, 
however, and several threats to validity remain. Prominent among these are 
the '’normality" problem and the "independence" problem (whether multiple 
effect sizes from a single study should be analyzed independently) . As a 
result, resampling methods have been proposed when it is assumed that 
distributions are nonnormal and multiple effect sizes are independent. 
However, resampling methods for nonnormal dependent multiple effect sizes 
have not been found. This paper discusses methods for resampling meta- 
analysis with dependent multiple effect sizes. First, the literature 
regarding the use of resampling for a univariate meta-analysis is reviewed. 
Then, a review of the independence problem (i.e., multiple effect sizes) is 
provided. Finally, resampling methods for countering the problems of 
nonnormality and nonindependence for the multivariate meta-analytic case are 
described. Educational researchers involved with meta-analysis are likely to 
find multiple effect sizes to be of issue. In most cases, if not all, 
multivariate methods will be preferred over univariate. Resampling methods 
can improve multivariate meta-analytic applications. (Contains 15 
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Abstract 

For the last decade, the use of meta-analytic methods in educational research has been 
widespread. Indeed, few aspects of education have escaped the meta-analytic revolution. 
However, the acceptance of findings has not been outright, as several validity threats remain 
debatable. Prominent among these are (a) the “normality” problem and (b) the “independence” 
problem (i.e., should multiple effect sizes from a single study be analyzed independently). 
Accordingly, resampling methods have been proposed when it is assumed that (a) distributions 
are non-normal and (b) multiple effect sizes are independent. Incidentally, however, resampling 
methods suitable for non-normal dependent multiple effect sizes have not been found. Herein, 
methods for resampling meta-analyses with dependent multiple effect sizes are discussed. First, 
the literature regarding the use of resampling for a univariate meta-analysis is reviewed. Second, 
a review of the “independence problem” (i.e., multiple effect sizes) is provided. Finally, 
resamphng-methodsdbrcounteringthe-problemsof— non-normality— and— non-independence— for 
the multivariate meta-analytic case are described. For educational researchers involved with 
meta-analysis, it is likely that multiple effect sizes will be of issue. In most cases (if not all), 
multivariate methods will be preferred over univariate. Moreover, resampling methods can 
improve multivariate meta-analytic applications. 
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Methods for Resampling Meta-Analyses with Multiple Effect Sizes 
In general, a biased estimate (statistic) is undesirable because it could lead one to 
misconclude and subsequently make ineffective decisions. In a meta-analytic review, effect size 
estimates may be grouped and cumulated to yield a “grand effect size” estimate. In this case, 
accumulation of bias is also possible. The importance of an unbiased effect size estimate is 
evident when one considers that a grand effect size is intended to provide a global decision point 
regarding a phenomena’s research base. Moreover, the scope of the meta-analytic applications 
(e.g., physical and social sciences) and range of interpretations (e.g., setting research priorities, 
establishing policies, drug certification) emphasize the need for improved bias control 
(modeling). Herein, resampling methods are explored as a means of controlling bias for 
multivariate non-normal effect size estimates. 

Validity Threats to Meta-Analytic Findings 

Independence Problem 

In the simplest, non-degenerative case of a standardized mean difference (treatment 
effects) meta-analysis, a study (unit of analysis) will contain one treatment group and one 
endpoint (measure), thereby, producing one effect size estimate (statistic). However, a study can 
contain multiple treatment groups for a single control group and (or) multiple endpoints 
(measures) for each dependent variable. In these cases, multiple estimates of effect size within a 
single study are possible. Consequently, to statistically combine (synthesize) effect sizes within 
and (or) among studies, one must decide whether the multiple effect sizes are stochastically 
independent Indeed, Rosenberg, Adams, and Gurevitch (2000) suggest that the assumption of 
independence “is an important and substantive issue for the person carrying out the analysis to 
think through with care” (p. 6). 
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Normality Problem 

To date, many approaches for modeling multiple effect sizes exist (see e.g., Hedges & 
Olkin, 1985 chap. 10; Glass, McGaw, & Smith, 1981, chap. 6; Gleser & Olkin, 1994; Kalaian & 
Raudenbush, 1996; Raudenbush, Becker, & Kalaian, 1988; Raudenbush & Bryk, chap. 7; 
Rosenthal & Rubin, 1986; Timm, 1999a, 199b). However, current multivariate models assume 
that the distribution of population effect sizes is multivariate normal (parametric modeling). 
Accordingly, in cases where the assumption of distributional normality is not met, parametric 
models can produce biased results. In general, resampling schemes can be used to approximate 
the distributions of statistics (to include multivariate analyses) under almost no distributional 
assumptions (thereby reducing bias of the parametric model). Indeed, resampling schemes for 
independent non-normal meta-analytic data have been advanced (see Adams, Gurevitch, & 
Rosenberg, 1997; Brown, Homer, & Inman, 1998). However, resampling applications for non- 
independent non-normal cases have not been found. Consequently, the following question is of 
interest: Are resampling methods applicable to modeling multivariate effect sizes? 

Rationale for Resampling Multiple Effect Sizes 

The term “resampling”, as used herein, collectively refers to (a) jackknifing, (b) 
bootstrapping, and (c) permuting. A review of the suitability of each method to meta-analytic 
data now follows: 

Jackknifing 

In general, the “jackknife” procedure creates new samples (i.e., resamples) by sequentially 
removing an observation from the original (observed) data. Glass, McGaw, and Smith (1981) 
proposed using the jackknife to estimate confidence intervals for the multiple effect size case. 
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Based on simulation results, they concluded that “the jackknife method appears to be appropriate 
and equal to the task of handling data sets interlaced with complicated dependencies.” (p. 208). 
Bootstrapping 

In general, the “bootstrap” procedure creates new samples by selecting an exact size 
simple random sample from the original (observed) data. According to Shao and Tu (1995) “the 
bootstrap provides a nonparametric alternative for approximating the distributions of statistics in 
multivariate analysis under almost no distributional assumptions” (p. 373). Accordingly, the 
bootstrap would seem an useful approach for meta-analytic data. However, applications of the 
bootstrap to multiple effect sizes could not be found. Consequently, the author is working on a 
bootstrap approach for both (a) testing homogeneity of effects sizes and (b) computing class 
(grand) confidence intervals. 

Permuting 

In general, a “permutation” procedure creates a new sample by randomly assigning 
subjects (in our case, effect sizes) to class levels. According to Good (2000), for a permutation 
test to be exact and unbiased the observations must be exchangeable. Furthermore, 
exchangeable observations in the case of dependence must also have normally distributed 
random variables. Based on this constraint, permuting does not seem to be a viable resampling 
approach for multiple effect sizes with a non-normal distribution. 

Research Agenda 

To improve the resampling of multiple effect sizes the following research agenda is 
proposed: 

1. Where applicable, do resampling methods (i.e., nonparametric approaches) offer an 
improvement over parametric approaches to modeling multivariate effect sizes? 
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2. Where more than one resampling method is applicable, how does each compare with regard to 
desirable modeling properties? 

3. Do diagnostics techniques (e.g., double bootstrapping) and computational enhancements (e.g., 
Monte Carlo simulation) improve resampling methods for the non-normal non-independent 
case? 

Conclusions 

Today, educational researchers must engage decades of primary research that is often 
interdisciplinary and from fields that continue to subspecialize. Consequently, meta-analytic 
studies are essential for cumulating findings and revealing new research opportunities. 
Resampling methods can improve these efforts. 
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